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Selecting  and  managing  development  project  portfolios  is  a  critical  function 
within  the  Air  Force’s  weapon  systems  development  process.  Decision  makers 
must  weigh  benefits,  costs,  and  mission  needs  for  a  variety  of  proposed  new 
initiatives  and  current  weapon  systems  programs  in  order  to  develop  an  effective 
portfolio  that  provides  the  best  value  to  the  User.  This  research  examines  current 
Air  Force  development  portfolio  management  practices  as  perceived  by  those 
involved  with  the  decision  making  process.  Research  findings  indicate  several 
gaps  between  Air  Force  practices  and  those  commercial  best  practices  as  found 
in  the  literature.  A  general  approach  for  bridging  the  differences  is  offered  as 
recommendations. 


With  an  annual  research,  develop¬ 
ment,  test,  and  evaluation 
(RDT&E)  budget  of  $37.9  bil¬ 
lion,  and  an  additional  $60.3  billion  allocated 
toward  weapon  systems  procurement  in 
fiscal  year  2001,  the  Department  of  De¬ 
fense  (DoD)  spends  more  on  weapon  sys¬ 
tems  development  and  procurement  than 
any  other  organization  in  the  world  spends 
on  new  product  development  (NPD) 
(DoD,  2000a). 

Of  this  $98.2  billion,  $9.7  billion  in 
RDT&E  funds  and  $14.1  billion  in 
procurement  funds  were  appropriated 


specifically  for  Air  Force  weapon  systems 
development  and  procurement.  While  stag¬ 
gering  at  first  glance,  today’s  weapon  sys¬ 
tems  development  budget  has  only  recently 
stabilized  after  more  than  a  decade  of  re¬ 
duction  in  spending. 

Measured  in  constant  fiscal  year  2001 
dollars,  DoD  RDT&E  spending  has 
decreased  nearly  18  percent  while  pro¬ 
curement  spending  has  been  cut  56  per¬ 
cent  over  the  last  16  years  (DoD,  2000b). 
This  constrained  resources  environment 
has  been  a  major  driver  behind  resurgent 
acquisition  reform  efforts.  Overarching  Air 
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Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 
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Force  efforts  include,  but  are  not  limited 
to,  the  introduction  of  acquisition  reform 
Lightning  Bolts,  the  implementation  of  ac¬ 
quisition  reform  initiatives  (Lean  Aero¬ 
space  Initiative  [LAI],  Cycle  Time  Reduc¬ 
tion,  Cost  as  an  Independent  Variable 
[CAIV],  Reduction  in  Total  Ownership 
Costs  [R-TOC],  and  Single  Process  Ini¬ 
tiative  [SPI]),  and  the  establishment  of 
reinvention  teams. 

Identifying  and  analyzing  commercial 
NPD  and  development  portfolio  manage¬ 
ment  best  practices  has  been  the  subject 
of  many  research  efforts  (Calantone  et  al, 
1995;  Cooper  et  al,  1995,  1998a,  1998b, 
1999,  2000;  Griffin,  1997;  Gupta  et  al, 
2000;  Lester,  1998;  Poolton  &  Barclay, 
1998;  Scott,  2000).  To  date,  these  efforts 
have  focused  solely  on  commercial 
industry  practices. 

The  objective  of  this  research  is  to,  for 
the  first  time,  shift  the  focus  from  the 
commercial  sector  to  that  of  military 
weapon  systems  development.  Specifi¬ 
cally,  this  research  examines  the  current 
state  of  Air  Force  development  portfolio 
management  processes,  where  the  devel¬ 
opment  portfolio  is  defined  as  a  grouping 
of  weapon  systems  development  projects 
by  either  a  common  functionality  or  a  com¬ 
mon  mission  area. 

Issues  of  Complexity 
and  Uniqueness 


In  the  weapon  systems  development 
environment,  the  operation  of  a  sound 
development  portfolio  management  pro¬ 
cess  is  imperative  and  is  an  endeavor  that 
comes  with  challenges  unique  to  those 
found  in  the  commercial  sector.  For  ex¬ 
ample,  economic  measures  including 


increased  profits,  revenue  generation, 
increased  market  share,  and  increased 
shareholder  value  represent  some  of  the 
more  common  measures  of  NPD  success 
in  commercial  industry. 

However,  success  in  weapon  systems 
development  cannot  easily  be  measured 
in  economic  terms  and  success  criteria  are 
often  more  qualitative  in  nature.  Other 
unique  aspects  of  weapon  systems  devel¬ 
opment  include  the  importance  and  devel¬ 
opment  of  military  strategy,  the  synergis¬ 
tic  nature  of  weapon  systems,  political 
oversight  of  the  process,  and  the  resource 
allocation  process.  These  issues  are  not 
mutually  exclusive;  instead,  there  is  sig¬ 
nificant  overlap  and  interaction  between 
these  four  areas  of  complexity  and  unique¬ 
ness.  For  example,  issues  of  political  over¬ 
sight  add  to  the  complex  nature  of  military 
strategic  planning  as  well  as  the  resource 
allocation  process.  The  following  com¬ 
ments  regarding  these  issues  are  presented 
with  an  Air  Force  perspective. 

Strategy 

Strategic  planning  is  a  key  element 
within  the  DoD.  Weapon  systems  devel¬ 
opment  plays  a  crucial  role  in  determining 
strategic  goals,  objectives,  and  the  direction 
of  U.S.  policy.  Successful  weapon  systems 
employment  provides  U.S.  leaders  with 
options  in  enforcing  both  domestic  and  in¬ 
ternational  policy.  The  ability  to  gauge 
today’s  environment,  both  domestic  and 
internationally,  and  predict  trends  10,  15, 
or  even  20  years  in  the  future  is  no  easy 
task.  Entire  directorates  exist  whose  sole 
charter  is  to  conduct  strategic  planning 
activities.  Strategic  plans  and  policies 
emerge  only  after  thorough  analysis  of  in¬ 
ternational  and  domestic  threats,  potential 
technological  breakthroughs,  and  determin- 
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Other 

Federal 

Agencies 


Other  Services: 
Navy,  Army, 
Marine  Corps 


National  Security  Strategy 

Promote  prosperity 
Promote  democracy 
Enhance  security 


National  Military  Strategy 

Shape  the  international  environment 
Respond  to  the  full  spectrum  of  crises 


Air  Force  Strategic  Plan 

•  Air  and  space  superiority 

•  Global  attack 

•  Rapid  global  mobility 

•  Precision  engagement 

•  Information  superiority 

•  Agile  combat  support 


Unified  \ 
Combatant  i 
Commands  ' 


Air  Force  Major 
Command  level 


Air  Force  Wing  level 


Figure  1.  National  Security  Strategy  Flow 


ing  the  presence  the  United  States  will 
have  on  global  affairs. 

The  hierarchical  flow  of  strategy  de¬ 
velopment  is  presented  in  Figure  1 .  At  its 
highest  level,  national  security  strategy  is 
developed,  which  flows  to  the  DoD.  At 
the  DoD  level,  a  joint  service  view  of  how 
to  support  the  national  security  strategy  is 
developed.  From  there,  each  service  is 
empowered  to  construct  strategic  plans 
and  policies  that  support  the  DoD  j oint  ser¬ 
vice  perspective,  eventually  providing  sup¬ 


port  to  the  national  security  strategy  (DoD, 

2000c). 

Figure  1  provides  insight  into  the  com¬ 
plex  nature  of  this  process.  Each  level  of 
strategic  planning  introduces  the  opportu¬ 
nity  for  differences  in  interpretation.  Ad¬ 
ditionally,  at  each  level,  the  coordination 
of  multiple  participants  becomes  increas¬ 
ingly  difficult.  For  example,  the  DoD  is 
not  alone  in  providing  support  of  the 
national  security  strategy.  Agencies  such 
as  the  Department  of  Energy,  the  Central 
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Intelligence  Agency,  and  the  Department 
of  State  are  all  involved  with  national 
security  issues.  At  lower  levels,  bringing 
together  the  Army,  Navy,  Marine  Corps, 
and  Air  Force  leadership  to  develop  syn¬ 
ergistic  plans  effectively  can  itself  become 
a  politically  charged  endeavor. 

Synergistic  Nature  of  Weapon  Systems 

Another  aspect  that  distinguishes 
weapon  systems  development  from  its 
commercial  counterparts  is  the  synergis¬ 
tic  nature  of 
these  systems. 
In  the  modem- 
era  of  warfare, 
never  has  one 
of  the  three 
components 
(air,  sea,  or 
land)  been  able 
to  achieve  pre- 
determined 
strategic  objec¬ 
tives  alone;  the 
most  recent 
examples  being  Operation  Desert  Storm 
and  the  conflict  in  former  Yugoslavia.  In 
both  cases,  the  ability  to  synergize  the  ca¬ 
pabilities  of  the  Army,  Navy,  Marine  Corps, 
and  Air  Force  provided  the  U.S.-led  coali¬ 
tion  with  the  overarching  capability  to  con¬ 
duct  overwhelmingly  dominant  activities 
against  its  adversaries. 

The  historic  portrayal  of  military  con¬ 
flict  as  a  two-dimensional  environment, 
where  generals  direct  the  war  using  a 
game  board  representation  of  the  battle¬ 
field,  is  long  outdated.  Today’s  battlefield 
is  an  orchestration  of  air,  ground,  sea,  and 
space  assets,  all  integrated  and  connected 
by  vast  communications  and  information 


dissemination  networks  that  form  the 
backbone  of  such  a  synergistic  battlefield. 
One  of  the  key  components  of  the  Joint 
Vision  2020  document  is  the  ability  for 
weapon  systems,  and  U.S.  forces  in  gen¬ 
eral,  to  integrate  with  each  other.  Joint 
Vision  2020  (DoD,  2000d)  defines 
interoperability  as  the  “ability  of  systems, 
units,  or  forces  to  provide  services  and 
accept  services  from  other  systems,  units, 
or  forces  and  to  use  the  services  so  ex¬ 
changed  to  enable  them  to  operate  effec¬ 
tively  together”  (p.  15). 

While  commercial  practices  lean  to¬ 
ward  using  common  components  and  com¬ 
mon  platforms,  the  requirement  to  integrate 
with  other  commercial  systems  is  much 
less  imperative  than  this  requirement  in  a 
weapon  systems  environment.  For  ex¬ 
ample,  if  Ford  discontinues  production  of 
the  Escort,  it  is  likely  to  have  no  effect  as 
to  how  Ford’s  remaining  portfolio  of  ve¬ 
hicles  will  perform.  However,  if  you  re¬ 
move  a  weapon  system  like  the  Milstar 
Satellite  Communications  System  from  the 
battlefield,  you  negatively  impact  nearly 
every  other  weapon  system  in  their  ability 
to  securely  communicate  with  each  other 
and  thereby  reduce  the  overall  capability 
of  these  systems. 

Political  Oversight 

One  of  the  more  unique  aspects  of 
weapon  systems  development  acquisition 
is  the  impact  congressional  oversight  can 
have  on  a  weapon  systems  acquisition 
program.  As  provided  for  in  Article  I  of 
the  Constitution,  Congress  controls  the  de¬ 
fense  authorization  and  appropriations  pro¬ 
cess.  Congress  wields  tremendous  power 
during  the  defense  authorization  and  ap¬ 
propriations  approval  process.  At  any  point 


"The  historic 
portrayal  of 
military  conflict  as 
a  two-dimensional 
environment,  where 
generals  direct  the 
war  using  a  game 
board  representation 
of  the  battlefield,  is 
long  outdated." 
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in  this  process,  Congress  can  cut,  add,  or 
leave  requested  funding  levels  unchanged 
as  submitted  in  the  president’s  budget. 

Decisions  to  cut  or  add  funds  to  spe¬ 
cific  weapon  systems  acquisition  programs 
can  have  profound  effects  throughout  the 
entire  DoD.  For  example,  if  Congress  de¬ 
cides  to  cut  $10  million  in  development 
funds  from  a  proposed  Air  Force  missile 
program,  decision  makers  must  then  back 
and  determine  the  effects  of  such  a  cut 
(the  ability  to  support  strategic  goals,  in¬ 
teraction  with  other  weapon  systems  [both 
Air  Force  and  other  services],  and  effects 
of  schedule  delay  and  increased  unit  costs) 


and  eventually  recommend  from  which 
programs  funds  will  be  taken. 

Congressional  actions  to  increase  fund¬ 
ing  levels  can  also  have  negative  implica¬ 
tions.  For  example,  use  the  same  Air  Force 
missile  development  program,  and  instead 
of  cutting,  Congress  adds  $10  million. 
While  this  situation  is  likely  beneficial  for 
the  missile  program,  typically  what  hap¬ 
pens  is  that  Congress  does  not  increase 
the  overall  Air  Force  budget  by  $  1 0  mil¬ 
lion,  therefore  forcing  tough  decisions  re¬ 
garding  from  which  other  development 
programs  to  take  the  $  10  million  in  order 
to  support  the  Congressional  request. 


•  Project 
evaluation 

•  Trade-offs 

•  Develop 
“corporate 
position” 


Figure  2.  AFCS  Decision-making  Process  Flow 
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Resource  Allocation 

The  resource  allocation  process  is  the 
focus  of  this  research,  and  therefore 
deserves  additional  introduction.  The  pri¬ 
mary  tool  used  by  the  DoD  in  its  resource 
allocation  activities  is  the  Planning,  Pro¬ 
gramming,  and  Budgeting  System  (PPBS; 
DoD,  1984).  The  PPBS  is  a  cyclic  pro¬ 
cess  that  provides  for  decision  making  on 
future  programs  and  permits  decisions  to 
be  examined  and  analyzed  from  the 
viewpoint  of  the  current  environment  — 
threat,  political,  economic,  technological, 
and  resources.  Of  particular  interest,  the 
multi-functional  Air  Force  Corporate 
Structure  (AFCS)  is  charged  with  making 
decisions  regarding  resource  allocation 

against  all  Air 
Force  activi¬ 
ties,  including 
weapon  sys¬ 
tems  acquisi¬ 
tion  programs. 
Figure  2  pro¬ 
vides  insight 
into  the  flow  of 
the  AFCS  and 
how  this  decision-making  body  fits  within 
the  PPBS  process. 

One  of  the  most  unique  aspects  of  this 
resource  allocation  decision-making  pro¬ 
cess  comes  in  the  form  of  restrictions 
placed  upon  the  funds  allocated  during  this 
process.  The  Department  of  Defense  Fi¬ 
nancial  Management  Regulation  (DoD, 
1993)  provides  guidance  on  the  use  of 
funds  for  weapon  systems  development. 
Within  the  DoD,  funds  are  divided  into 
separate  categories  commonly  known  as 
colors  of  money.  For  example,  within  the 
Air  Force  there  are  five  different  catego¬ 
ries,  or  colors,  that  correspond  to  a  par¬ 
ticular  activity  during  the  weapon  systems 


development  process.  These  categories 
and  codes  include:  RDT&E  (3600),  air¬ 
craft  procurement  (30 1 0),  ammunition  pro¬ 
curement  (3011),  missile  procurement 
(3020),  and  other  procurement  (3080). 
Once  a  weapon  system  is  fielded,  opera¬ 
tions  and  maintenance  (3400)  funds  are 
allocated  for  support  of  the  system. 

In  addition  to  the  restrictions  placed  upon 
the  use  of  funds,  funds  are  restricted  in 
their  lifespan.  This  means  that  funds  have 
to  be  obligated  against  contractual  agree¬ 
ments  within  a  limited  amount  of  time. 
However,  this  restriction  does  not  mean 
the  work  has  to  be  completed  within  this 
limited  time  frame.  For  weapon  systems 
development,  these  limits  range  from  two 
years  for  RDT&E  funds,  to  three  years 
for  procurement  funds.  For  example,  if  the 
Air  Force  has  a  contract  with  Boeing  to 
build  aircraft,  the  Air  Force  would  use  air¬ 
craft  procurement  (30 1 0)  funds  that  would 
have  to  be  obligated  against  the  contract 
within  three  years  of  the  fiscal  year  of 
appropriation.  Because  of  the  complex 
technical  nature  of  weapon  systems  ac¬ 
quisition,  at  any  given  snapshot  in  time,  a 
weapon  systems  program  may  be  obligat¬ 
ing  both  RDT&E  funds  and  procurement 
funds  that  span  a  wide  range  of  useable 
lifespan. 

Past  Research  About  Commercial  NPD 
and  Development  Portfolio 
Management 


Identifying  and  analyzing  commercial 
NPD  and  development  portfolio  man¬ 
agement  best  practices  has  been  the 
subject  of  many  research  efforts.  While  a 
variety  of  methodologies  have  been 
employed  in  conducting  these  research 


"The  primary  tool 
used  by  the  DoD  in 
its  resource  alloca¬ 
tion  activities  is  the 
Planning,  Program¬ 
ming,  and  Budgeting 
System  (PPBS)" 
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efforts,  a  common  theme  runs  throughout 
—  identifying,  analyzing,  and  eventually 
implementing  commercial  NPD  and  de¬ 
velopment  portfolio  management  best 
practices  can  improve  an  organization’s 
NPD  process.  Poolton  and  Barclay  (1998) 
and  Griffin  ( 1 997)  took  historical  perspec¬ 
tives  in  identifying  commercial  NPD  best 
practices.  Poolton  and  Barclay  (1998) 
identified  eight  major  variables  and  an  ad¬ 
ditional  nine  key  strategic  factors  found  to 
be  critical  in  successful  commercial  NPD 
efforts.  Griffin’s  (1997)  research  focused 
on  updating  the  1990  Product  Develop¬ 
ment  and  Management  Association 
(PDMA)  survey  of  commercial  NPD  best 
practices. 

The  results  indicated  practices  such  as 
operating  under  a  formal  NPD  process, 
developing  an  NPD  strategy  and  tying  that 
strategy  into  overall  organization  objectives, 
and  implementing  a  multi-functional  ap¬ 
proach  are  still  important  factors  for  im¬ 
proving  NPD  success.  Calantone  et  al. 
(1995)  developed  a  list  of  40  fundamental 
principles  for  commercial  NPD  and  mea¬ 
sured  the  level  of  agreement  between 
R&D  and  engineering  practitioners  and 
marketing  professionals.  Research  findings 
showed  a  strong  overall  level  of  agree¬ 
ment  between  the  two  groups.  Lester 
(1998)  identified  16  factors  within  five 
overarching  areas:  senior  management 
commitment,  organization  structure,  at¬ 
tractive  new  product  concepts,  venture 
teams,  and  project  management. 

Gupta  et  al.  (2000)  found  that,  while 
both  “high-R&D  effective”  and  “low- 
R&D  effective”  organizations  had  similar 
views  on  the  importance  of  certain  com¬ 
mercial  NPD  success  factors,  it  is  the 
“high-R&D  effective”  firms  that  were 
more  capable  of  implementing  these 


practices.  Scott  (2000)  took  a  different 
approach  and  through  the  use  of  the 
DELPHI  Issues  Methodology  ranked  the 
24  most  critical  problems  facing  high-tech 
commercial  NPD  firms.  The  top  five 
included:  strategic  planning,  project  selec¬ 
tion,  organizational  learning,  core  compe¬ 
tencies,  and  cycle  time  reduction.  These 
research  efforts  will  be  later  used  to 
develop  a  framework  of  commercial  best 
practices  against  which  the  findings  of  this 
research  will  be  measured.  As  such,  the 
results  of  past  research  will  be  discussed 
in  more  detail. 

Perhaps  some  of  the  most  comprehen¬ 
sive  research  in  the  field  of  commercial 
NPD  and  devel¬ 
opment  portfolio 
management 
has  come  from 
the  trio  of  Coo¬ 
per,  Edgett,  and 
Kleinschmidt 
(1995,  1998a, 

1998b,  1999, 

2000).  Much  of 
their  work  has  centered  on  conducting 
extensive  analysis  of  best  practices  within 
a  vast  spectrum  of  commercial  sectors. 
This  group  of  researchers  has  been  at  the 
forefront  of  the  movement  to  view  the 
selection  and  management  of  development 
projects  as  portfolios,  much  like  portfolios 
are  viewed  in  a  financial  and  investment 
construct. 

Additionally,  their  research  has  resulted 
in  the  development  of  the  stage-gate 
process  that  calls  for  a  management  phi¬ 
losophy  that  is  grounded  in  the  belief  that 
difficult  go/kill  decisions  must  be  made 
at  select  points  within  the  NPD  process. 
Cooper  et  al.  (1998b)  provided  a  concise 
and  comprehensive  list  of  practices 


"Much  of  f  heir  work 
has  centered  on 
conducting  extensive 
analysis  of  best 
practices  within  a 
vast  spectrum  of 
commercial  sectors." 
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required  for  an  effective  development  port¬ 
folio  management  process. 

Research  Methodology 


Focusing  on  the  exploratory  nature  of 
this  research,  a  structured  interview  (see 
Appendix)  was  developed  in  order  to  col¬ 
lect  data  and  establish  a  sound  framework 
for  future  analysis  regarding  the  effective¬ 
ness  of  Air  Force  development  portfolio 
management  practices.  The  structured  in¬ 
terview  contained  both  open-ended  and 
closed-ended  questions.  Open-ended  ques¬ 
tions  allowed 
the  respondent 
the  opportunity 
to  provide  in- 
depth  insight 
and  discuss  a 
topic  in  a  more 
unstructured 
format,  while  closed-ended  questions  pro¬ 
vided  a  methodology  for  confirming  com¬ 
mon  themes  emerging  from  the  open-ended 
questions. 

A  total  of  seven  closed-ended  questions 
were  included  in  this  structured  interview 
and  were  answered  using  a  five-point 
Likert  scale,  anchored  by  “strongly  dis¬ 
agree”  (1)  to  “strongly  agree”  (5)  and  a 
neutral  midpoint  of  “uncertain”  (3).  The 
exception  to  this  scale  is  the  question  re¬ 
garding  the  use  of  decision  support  tools, 
which  was  anchored  by  “less  than  20  per¬ 
cent”  (1)  to  “more  than  80  percent”  (5) 
with  20  percentage  point  increments  in 
between.  Prior  to  administering  the  struc¬ 
tured  interview,  a  pilot  study  was  con¬ 
ducted  with  a  group  of  researchers  knowl¬ 
edgeable  in  the  area  of  commercial  NPD 
and  development  portfolio  management. 


This  exercise  led  to  several  small  re¬ 
finements  and  the  addition  of  several 
questions. 

The  structured  interview  package  in¬ 
cluded  a  brief  discussion  detailing  the 
research  objectives,  purpose  of  the  inter¬ 
view  process,  and  bounds  of  the  structured 
interview.  The  structured  interview  itself 
was  divided  into  several  parts,  with  a  total 
of  24  questions.  The  first  part  contains 
questions  designed  to  collect  basic  demo¬ 
graphic  information  regarding  the  respon¬ 
dent.  The  next  section  includes  questions 
focused  on  the  strategic  alignment  of  Air 
Force  weapon  systems  development 
projects.  Additional  sections  pose  questions 
in  the  areas  of  organizational  communica¬ 
tion,  use  of  decision  support  tools,  feed¬ 
back  processes,  identification  of  risk  and 
uncertainty,  determination  of  weapon  sys¬ 
tem  value,  and  identification  of  the  effects 
of  schedule.  A  final  section  allows  for  the 
interview  subject,  in  an  open-ended  for¬ 
mat,  to  identify  additional  strengths  and 
weaknesses  in  the  Air  Force’s  weapon  sys¬ 
tems  development  portfolio  management 
process  not  previously  discussed. 

The  sample  consisted  of  25  respondents 
who,  at  the  time  of  the  data  collection 
process,  were  involved  with  the  AFCS 
decision-making  process.  The  sample  size, 
while  relatively  small,  provides  a  represen¬ 
tative  view  of  the  AFCS  community.  Re¬ 
spondents  included  Panel  and  IPT  Chair¬ 
persons,  Panel  and  Integrated  Process 
Team  (IPT)  members  from  a  cross-section 
of  weapon  systems  platforms  and  func¬ 
tional  areas,  Group  and  Board  members, 
and  other  senior  Air  Force  weapon  sys¬ 
tems  procurement  policy  makers.  Respon¬ 
dents  were  categorized  based  on  three  de¬ 
mographic  areas:  function  within  the 
AFCS,  primary  AFCS  level  exposure,  and 


"The  structured 
interview  itself  was 
divided  into  several 
parts,  with  a  total  of 
24  questions." 
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Table  1 .  Respondent  Sample  Category  Breakout 


Function 

Exposure  Level 

Years  Experience 

Decision  Support 

Panel/IPT 

<1  Year 

(14)  -56% 

(8)  -  32% 

(6)  -  24% 

Decision  Maker 

Group  and  Higher 

1  -2  Years 

(7)  -  28% 

(3)  -  1 2% 

(6)  -  24% 

Policy  Maker 

Multi-level 

2+  Years 

(4)  -  1 6% 

(14)  -56% 

(13)  -52% 

years  experience  with  the  AFCS  decision¬ 
making  process. 

Table  1  provides  summary  information 
regarding  the  breakout  of  these  catego¬ 
ries.  Decision  makers  were  defined  as 
those  who  were  members  of  the  Group, 
Board,  or  Council,  or  those  who  were 
Panel  or  IPT  Chairpersons.  Decision  sup¬ 
port  respondents  primarily  consisted  of 
Panel  and  IPT  members,  and  a  cross-sec¬ 
tion  of  weapon  system  Program  Element 
Monitors  (PEMs).  Policy  makers  were 
defined  as  respondents  who,  while  actively 
involved  with  the  AFCS,  were  primarily 
focused  on  the  development  and  refine¬ 
ment  of  weapon  systems  acquisition 
policy.  Finally,  it  should  be  noted  that  nearly 
half  the  respondents  have  less  than  two 
years  of  experience  with  the  AFCS,  which 
is  typical  of  military  organizations  where 
service  members  rotate  positions  fre¬ 
quently. 

To  ensure  a  uniform  approach  to  the  data 
collection  process,  a  single  interviewer  con¬ 
ducted  the  structured  interview.  Interviews 
lasted  between  45  minutes  and  two  hours 
and  to  increase  data  reliability,  when  secu¬ 
rity  conditions  allowed,  the  interviews  were 
audiotaped  and  later  transcribed.  Data  analy¬ 


sis  focused  on  these  195  pages  of  transcrip¬ 
tions.  Two  independent  researchers  re¬ 
viewed  each  transcript  using  codes  and 
marginal  remarks  to  identify  connections  and 
common  issues  within  the  individual  inter¬ 
view  and  between  respondents  (Miles  & 
Huberman,  1 994).  At  the  completion  of  the 
independent  reviews,  both  assessments 
were  brought  together  and  findings  were  dis¬ 
cussed  and  eventually  synthesized. 

It  should  be  noted  that  the  data  analysis 
effort  revealed  some  interesting  results 
regarding  responses  to  the  scaled  response 
questions.  In  many  cases,  a  respondent’s 
assessment  to  a  scaled  response  question 
was  based  on  their  agreement,  or  disagree¬ 
ment,  with  only  a  portion  of  the  statement. 
For  example,  a  respondent  would  “agree” 
that  communication  within  the  same  evalu¬ 
ation  groups  were  effective,  but  would 
“disagree”  that  communication  between 
evaluation  groups  were  effective,  there¬ 
fore  leading  to  a  misrepresentation  in  their 
final  assessment  of  the  entire  statement. 
Due  to  the  number  of  cases  in  which  this 
communication  disconnect  seemed  to  oc¬ 
cur,  the  following  discussion  regarding 
research  results  is  limited  to  the  open- 
ended  questions. 
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Research  Results 


The  following  sections  address  each  of 
the  structured  interview  categories  — 
strategic  alignment,  organizational  commu¬ 
nication,  decision  support  tools,  feedback 
processes,  identification  of  risk  and  uncer¬ 
tainty,  determination  of  weapon  system 
value,  and  identification  of  the  effects  of 
schedule  —  and  discussion  focuses  on 
common  themes  discovered  within  the 
transcripts  during  the  data  analysis  pro¬ 
cess.  Additional  concepts  of  interest  com¬ 
plete  the  research  results  section. 

Structured  Interview  Categories 

Strategic  Alignment.  Data  analysis 
indicates  that  while  most  respondents  un¬ 
derstand  how  strategy  is  developed  and 
disseminated  through  the  AFCS,  more  than 
half  (56  percent)  of  those  interviewed  be¬ 
lieve  there  is  no  formal  process  in  place  to 
ensure  weapon  systems  development 
projects  are  aligned  with  strategic  goals 
and  objectives.  Nearly  half  (48  percent) 
of  those  interviewed  cite  documents,  in¬ 
cluding  the  Defense  Planning  Guidance 
(DPG),  the  Air  Force  Annual  Planning  and 
Programming  Guidance  (APPG),  the  Air 
Force  Strategic  Plan,  and  Joint  Vision  2020, 
as  primary  sources  of  strategic  guidance 
in  the  decision-making  process.  Members 
of  the  AFCS  are  briefed  and  review  all 
strategy  documents  prior  to  implementa¬ 
tion.  One  respondent  related: 

Documents  like  the  DPG,  APPG, 
vision  statements,  etc.  all  provide 
information  as  to  where  we  are 
heading.  The  AFCS,  the  Panels 
and  such,  are  all  involved  with  the 
review  and  approval  process  for 
these  documents,  so  I  think  they 


are  pretty  educated  as  to  where 
the  Air  Force  and  DoD  are  head¬ 
ing  with  strategy. 

Flowever,  the  perception  among  13  (52 
percent)  of  the  respondents  is  that  there 
are  no  formal  ties  that  link  projects  under 
consideration  back  to  strategic  goals  and 
objectives.  A  decision  maker  responded: 

Seems  to  be  big  disconnects  with 
what  we  are  supposed  to  work  on, 
what’s  our  focus,  and  what  should 
our  thrusts  be.  Doesn’t  seem  to 
be  any  links  to  the  Air  Force  vi¬ 
sion. 

A  respondent  involved  with  the  decision 
support  aspect  of  the  AFCS  stated: 

I  would  say  the  processes  that  are 
in  place  are,  arguably,  fragmented 
in  how  we  align  individual  pro¬ 
grams  and  projects  with  what’s 
written  in,  for  example  Air  Force 
and  DoD  strategic  planning... I 
would  say  there  is  a  lack  of  some 
kind  of  front-end  analysis,  front- 
end  assessment  that  says,  “OK, 
where  and  how  does  this  fit  and 
align  with  the  (strategy)?” 

Organizational  Communication.  Un¬ 
der  the  umbrella  of  organizational  commu¬ 
nication,  several  questions  were  posed 
covering  subjects  such  as  how  strategy 
was  communicated  within  the  AFCS,  what 
information  was  made  available  to  deci¬ 
sion  makers  regarding  development 
projects  under  consideration,  was  the  in¬ 
formation  based  on  subjective  assessments 
or  objective  analysis,  and  how  do  projects 
enter  the  AFCS  decision-making  process. 
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When  describing  what  information  In  describing  their  concern  over  the  lim- 

was  made  available  during  the  decision-  ited  information  the  RAPIDS  sheet  pro¬ 
making  process,  several  common  themes  vides,  a  respondent  described: 
emerged.  With  the  exception  of  large- 

scale  weapon  systems  (large-scale  pro-  Typically,  we  have  to  put  together 

grams  versus  small  programs  will  be  a  couple  of  lines  worth  of  impact 

addressed  more  in-depth  later),  most  de-  statements. . ..  So  you  try  and  cap- 

cisions  are  based  on  limited  information  ture  in  one  line  what  the  essence 

and  the  time  frame  in  which  these  deci-  of  the  impact  is  and  you  try  and 

sions  are  made  is  compressed.  Ten  re-  put  it  in  terms  of  operational  im- 

spondents  (40  percent)  expressed  con-  pact.  Sometimes  it’s  pretty  far 

cern  regarding  the  issue  of  limited  in-  fetched.  For  example,  “being  able 

formation,  while  an  additional  seven  re-  to  conduct  an  air  war’’  doesn’t 

spondents  cited  concern  over  the  short  provide  much  detail, 

decision-making  time  frame. 

The  Resource  Allocation  Programming  Other  phrases  that  captured  the  is- 
Information  Decision  System  (RAPIDS)  sue  of  limited  decision-making  informa- 
database  is  the  primary  tool  used  for  pre-  tion  included:  “pretty  terse  statements,” 
senting  information  to  AFCS  decisionmak-  “as  you  get  higher  up  in  the  process,  the 

ers.  The  RAPIDS  database  provides  a  depth  of  information,  the  volume,  gets  a 
single  sheet  of  information  for  each  lot  smaller,”  and  “depending  on  where 

project.  Information  typically  includes  the  they  sit,  they  may  not  have  much  infor- 

estimated  cost  for  the  program  in  each  mation  on  cost,  schedule,  and  perfor- 
fiscal  year,  proposed  quantities  produced  mance  available  to  them.” 
in  each  fiscal  year,  deltas  for  both  budget  Comments  from  a  decision  maker 
and  quantities  if  funds  are  being  taken  or  within  the  AFCS  highlighted  the  issue 
increased,  the  impacts  associated  with  the  of  short  decision-making  time  frames 

development  project,  and  where  the  project  and  the  negative  impacts  that  it  can  have 
ranks  on  other  priority  lists.  Additional  in-  on  a  portfolio: 
sight  into  the  RAPIDS  database  is  pro¬ 
vided  by  a  respondent  who  stated:  What  happens  is  that  budget  bo¬ 

geys,  or  budget  targets,  are  given 
Within  a  RAPIDS  slide,  which  out  and  you  are  told  that  you  have 

captures  issues  the  panel  takes  a  24  hours  to  come  up  with  an  $8 

look  at  and  gets  briefed  up  through  million  cut  to  your  program.  We 

the  process.  Within  that  informa-  peanut  butter  spread  it  around, 

tion  it  has  proposed  funding 

changes  and  text  associated  with  Additional  insight  is  provided  as  to  the 

that.  The  very  first  line  within  the  extreme  nature  of  timing  and  decision- 

impacts  part  of  that,  the  individual  making  within  the  AFCS : 

has  to  state  how  that  particular 
project  is  reflected  in  the  DPG. 
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We  are  on  such  a  fast  track  to  make 
the  “call”  that  we  may  not  be  able 
to  get  all  the  information  needed  to 
assess  the  impacts. ...  If  the  guy  at 
the  Air  Staff  needs  to  make  a  call 
and  can’t  get  in  touch  with  some¬ 
one  at  the  program,  then  they  have 
to  make  a  guess  with  their  “best- 
guess-windage.” 


As  a  follow-up  to  the  question  regard¬ 
ing  decision-making  information,  respon¬ 
dents  were  asked  to  discuss  whether  they 
perceived  the  information  provided  to 
decision  makers  to  be  based  more  on 
subjective  assessments  or  based  more  on 
objective  analysis.  Fourteen  (56  percent) 
of  those  interviewed  perceived  the  infor¬ 
mation  to  be  more  subjective,  while  seven 
respondents  categorized  the  information  as 
a  combination  between  subjective  and  ob¬ 
jective.  A  decision  maker  described  the 
dual  nature  of  information,  stating: 


I  think  there  is  a  lot  of  analysis  to 
support  a  particular  position,  to  try 
and  come  to  some  truth  as  to 
where  they  stand.  Then  a  lot  of 
times  there  are  decisions  that  have 
to  be  made  that  are  not  necessar¬ 
ily  (objective),  you  couldn’t  put 
them  into  a  cookie  cutter,  and  you 
have  to  make  some  subjective  as¬ 
sessments  based  on  the  goals  and 
objectives. 


Respondents  were  asked  to  assess  to 
what  extent  they  perceive  new  develop¬ 
ment  projects  entering  the  process  through 
the  more  structured  bottom-up  approach 
or  those  that  are  directed  from  senior  lead¬ 
ership.  Several  themes  emerged.  First,  the 
majority  of  those  interviewed  (60  percent) 


perceived  that  most  new  development 
projects  enter  the  AFCS  decision-making 
process  through  the  more  structured  bot- 
tom-up  approach.  Seventeen  respondents 
(68  percent)  said  that  anywhere  between 
60  percent  and  95  percent  of  all  new 
development  projects  are  bottom-up 
initiatives. 

Even  though  respondents  indicated  a 
majority  of  projects  enter  the  AFCS 
through  the  bottom-up  process,  several 
respondents  expressed  the  need  to  secure 
senior  leadership  support  in  order  to  in¬ 
crease  the  likelihood  for  funding  alloca¬ 
tion.  A  decision  maker  in  the  AFCS  pro¬ 
cess  said: 


More  successful  are  those  that 
come  through  the  structured  pro¬ 
cess  —  bottom-up.  Part  of  the 
process  is  you  work  it  up  and  get 
acceptance  at  a  high  level  and  then 
you  are  allowed  to  do  it.  It  makes 
your  life  a  lot  easier  with  high- 
level  support. 


Finally,  six  respondents  (24  percent)  dis¬ 
closed  some  concern  regarding  the  poten¬ 
tial  for  development  projects  to  enter  the 
AFCS  process  through  multiple  points.  A 
decision  maker  who  has  experienced  this 
recalled: 


We  fight  through  the  15 -month 
budget  cycle;  defend  every  dol¬ 
lar;  go  through  a  lot  of  cut  drills 
and  agony.  Then  14-and-a-half 
months  into  the  process,  someone 
comes  up  with  a  neat  idea  and  it 
gets  briefed  to  a  senior  leader. 
They  like  it  and  the  hunt  for  money 
is  on.  Usually,  money  is  taken 
from  other  programs. 
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Decision  Support  Tools.  In  the  con¬ 
text  of  this  research,  decision  support  tools 
were  defined  as  those  products  or  pro¬ 
cesses  that  provide  the  decision  maker  with 
a  structured  and  interactive  approach  to 
accessing  and  interpreting  data,  with  a  pri¬ 
mary  goal  of  increasing  the  objectivity 
within  the  decision  making  process.  Sev¬ 
eral  questions  were  asked  to  gain  insight 
into  how  decision  support  tools  were  imple¬ 
mented  within  the  AFCS  decision-making 
process;  what,  if  any,  tools  are  currently 
being  used  and  are  they  used  to  develop 
priority  rankings  or  to  gain  insight  into  in¬ 
dividual  projects. 

Data  analysis  revealed  several  clear 
themes.  First,  in  nearly  unanimous  fashion 
(96  percent),  respondents  perceived  that 
decision  support  tools  are  not  used  in  the 
AFCS  decision-making  process.  As  dis¬ 
cussed  earlier,  tools  such  as  the  RAPIDS 
database  are  used  to  organize  and  present 
project  information  but  are  not  designed 
to  aid  the  decision  maker  in  structuring  the 
decision-making  process.  Several  com¬ 
ments  reinforced  this  lack  of  decision  sup¬ 
port  tool  use  within  the  AFCS: 

Decision  support  tools  —  there 
are  none.  It’s  whatever  informa¬ 
tion  you  can  get  your  hands  on  and 
how  you  use  it  and  how  it’s  pre¬ 
sented. 

As  far  as  the  AFCS,  I’m  not  sure 
they  are  using  some  of  these  tools. 
Nothing  that  helps  in  the  struc¬ 
turing  of  the  decision-making 
process. 

I  don’t  think  we  have  good  ones. 

I  don’t  think  there  really  are  (good 
tools),  it’s  what  drives  you  more 


to  the  subjectivity  on  certain 
things.  Because  there  is  a  lack  of 
tools  to  systematically  step  one 
through  a  decision  or  to  do  it 
consistently. 

They  view  RAPIDS  as  a  decision 
support  tool,  and  it’s  just  not.  No 
analysis  provided....  We  make 
decisions  on  programs  and  we 
don’t  know  what  the  hell  it  does. 

We  come  to  forks  in  the  road  and 
we  take  both. 

When  asked  to  identify  whether  the 
tools  in  place  were  used  for  developing 
ranking  priority  lists  or  for  gaining  insight 
into  particular  development  projects,  there 
appears  to  be  some  variation.  While  not 
identified  as  a 
decision  support 

aid,  respondents  "Data  analysis 

addressed  to  revealed  several 

what  extent  the  clear  themes." 

RAPIDS  infor¬ 
mation  database 

was  used.  Eight  respondents  (32  per¬ 
cent)  thought  that  the  RAPIDS  infor¬ 
mation  slides  were  used  to  generate  pri¬ 
ority-  ranking  lists,  while  an  additional 
eight  (32  percent)  thought  the  informa¬ 
tion  provided  by  the  database  was  used 
to  gain  insight  into  the  development 
projects.  Still,  another  eight  (32  percent) 
of  those  interviewed  were  adamant  that 
the  since  the  RAPIDS  system  was  not 
a  decision  support  tool,  that  neither  rank¬ 
ing  lists  nor  project  insight  was  gained 
through  this  process. 

Feedback  Processes.  Questions  within 
this  section  were  designed  to  better  under¬ 
stand  which  AFCS  reviews  a  development 
program  encounters  once  it  has  received 
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initial  funding  and  is  considered  an  active 
project.  Data  analysis  indicates  some 
variation,  including  the  identification  by 
several  respondents  of  multiple  review 
paths.  The  most  prevalent  are  the  program 
reviews  conducted  by  the  Program  Ex¬ 
ecutive  Officer  (PEO)  function.  There  are 
seven  PEO  offices  within  the  Air  Force, 
each  responsible  for  a  portfolio  of  pro¬ 
grams  organized  under  an  overarching 
weapon  systems  category  (i.e.,  airlift  and 
trainers,  space,  or  command  and  control). 
These  reviews  are  conducted  on  a  regu¬ 
lar  periodic  basis. 

Within  the  AFCS  itself,  a  management 
by  exception 
process  was 
the  recurring 
theme  discov¬ 
ered  in  the  data 
analysis  pro¬ 
cess.  Seven  re¬ 
spondents  (28 
percent)  de¬ 
scribed  a  pro¬ 
cess  where  on¬ 
going  development  projects  only  return  to 
the  AFCS  for  review  when  the  program  is 
believed  to  be  in  trouble.  One  decision 
maker  said: 

Remember,  the  AFCS  only  re¬ 
views  the  deltas  to  a  program.  So, 
once  it  makes  a  decision  and  gives 
you  a  certain  amount  of  money  and 
direction  to  go  do,  it  never  goes  back 
and  reevaluates  your  effort  until 
you  either  need  more  money  or 
are  willing  to  give  some  up. 


the  AFCS  decision-making  process  and 
determining  how  those  risks  and 
uncertainties  were  assessed.  For  the  pur¬ 
pose  of  this  research,  uncertainty  was 
defined  as  the  likelihood  of  an  event  (either 
negative  or  positive)  occurring  (i.e.,  a  70 
percent  chance  a  weapon  systems  devel¬ 
opment  project  will  encounter  a  cost  over¬ 
run),  while  risk  was  defined  as  the  nega¬ 
tive  consequences  associated  when  such 
an  event  occurs  (i.e.,  when  a  development 
project  encounters  a  cost  overrun,  the  po¬ 
tential  effects  on  development  schedule 
and  system  performance). 

Seven  respondents  (28  percent)  identi¬ 
fied  funding  instability  as  an  uncertainty 
within  the  process,  while  an  additional  five 
respondent’s  (20  percent)  comments  point 
to  concerns  regarding  the  large  volume  of 
projects  being  reviewed  by  the  AFCS 
decision-making  body.  It  should  also  be 
noted,  that  although  previously  discussed, 
the  issues  of  limited  decision-making 
information  and  a  compressed  decision¬ 
making  time  frame  were  highlighted  as 
additional  sources  of  risk  and  uncertainty 
by  six  respondents. 

A  decision  maker  described  the  concern 
for  unstable  funding  streams,  stating: 

We  change  budgets  so  dramati¬ 
cally.  We  don’t  give  a  program  a 
budget,  sit  back,  and  let  them 
execute.  We  are  always  trying  to 
change  budgets  to  account  for  real 
world  contingencies  (i.e.,  Bosnia) 
or  another  program  is  overrunning 
costs  and  finding  money  to  fix  that; 
we  don’t  have  a  lot  of  stability. 


"Within  the  AFCS 
itself,  a  management 
by  exception  process 
was  the  recurring 
theme  discovered  in 
the  data  analysis 
process." 


Risk  and  Uncertainty.  The  two  ques-  He  goes  on  to  discuss  the  effects  of 
tions  in  this  section  focused  on  identifying  budget  uncertainty  on  a  development  pro¬ 
file  sources  of  risk  and  uncertainty  within  gram.  He  described: 
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What  we  do  when  a  program  runs 
into  problems,  the  first  response 
is  to  cut  capability.  After  a  couple 
of  go-arounds,  the  User  comes 
back  and  says,  “If  you  cut  any 
more  capability  out  of  this  pro¬ 
gram,  we  don’t  want  it  anymore.” 

At  this  point  we  have  defined  the 
minimum  requirements.  Then 
something  else  happens  and  we 
need  to  slip  the  schedule  a  year 
or  two.  The  end  result  is  a  sys¬ 
tem,  which  has  less  capability  than 
the  User  originally  wanted,  that 
was  delivered  late. 

While  several  different  factors  emerged 
as  sources  of  risk  and  uncertainty  within 
the  AFCS  decision-making  process,  it  was 
clear  that  most  respondents  were  unaware 
of  any  process  within  the  AFCS  that  pro¬ 
vided  risk  and  uncertainty  assessments. 
Fifteen  respondents  (60  percent)  cited 
either  they  were  not  aware  of  such  a  pro¬ 
cess  or  that  they  believed  this  responsibil¬ 
ity  fell  upon  the  program  manager.  A 
respondent  stated: 

Within  the  AFCS  I  don’t  think 
there  are  any.  At  the  program  of¬ 
fice  level,  risk  and  uncertainty  es¬ 
timates,  those  are  all  done  at  the 
program  office.  That  is  totally 
external  to  the  AFCS.  This  infor¬ 
mation  doesn’t  get  relayed  up  to 
the  AFCS,  only  to  other  program 
reviews. 

Weapon  Systems  Value.  As  previ¬ 
ously  mentioned,  a  weapon  system’s  value 
historically  is  not  measured  in  economic 
terms  like  those  in  the  commercial  sector. 
The  questions  in  this  section  were  devel¬ 


oped  to  gain  additional  insight  into  how  the 
AFCS  measures  weapon  systems  value. 
Several  recurring  themes  emerged  during 
the  data  analysis  process.  As  far  as  what 
is  considered  to  be  a  measure  of  value, 
ten  respondents  (40  percent)  perceived 
weapon  systems  value  to  be  measured  by 
the  requirements  fulfilled  by  a  particular 
system,  while  an  additional  six  respondents 
(24  percent)  view  the  capabilities  a 
weapon  system  provided  as  the  primary 
measure  of  value  for  AFCS  decision  mak¬ 
ers.  Comments  used  to  describe  these 
measures  of 
value  included: 

“really  based  on 
the  ability  to  pro¬ 
vide  a  capabil¬ 
ity,”  “what  was 
the  capability 
provided  to  the 
User  —  that  is 
the  primary 
measure  of 
value,”  “based  on  the  vision,  how  does  the 
project  match  up  against  those  capabili¬ 
ties,”  and  “definitely  the  warfighters  value 
and  the  requirements.” 

The  data  also  revealed  that  even  though 
there  are  perceived  measures  of  weapon 
system  value,  the  AFCS  decision-making 
activities  do  not  include  a  consistent  pro¬ 
cess  of  quantifying  or  actually  measuring 
a  weapon  system’s  value.  Eleven  respon¬ 
dents  (44  percent)  reported  that  there  were 
no  techniques  or  processes  for  measuring 
value,  and  an  additional  five  respondents 
(20  percent)  discussed  the  subjective 
manner  in  which  value  was  determined. 
In  addressing  the  issue  of  no  techniques 
or  processes  for  measuring  value,  a 
respondent  cited: 


"As  previously 
mentioned,  a  weapon 
system's  value 
historically 
is  not  measured 
in  economic  terms 
like  those  in  the 
commercial  sector." 
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I’m  not  aware  of  any,  believe  it  or 
not.  I’ve  never  seen  anything  like 
that,  and  I’ve  seen  a  complete 
cycle.  It’s  subjective,  what’s  on 
the  list,  the  Chief’s  (Air  Force 
Chief  of  Staff)  list  is  a  big  thing.  I 
never  see  anything  quantitative. 

Other  comments  included:  “usually 
based  on  a  subjective  feeling,”  “mostly 
based  on  subjective  assessment,”  and  “I 
can’t  recall  value  being  addressed  too 
much.” 

Effects  of  Schedule.  Questions  here 
again  focused  on  determining  whether  or 
not  techniques  or  processes  were  in  place 
to  ensure  AFCS  decision  makers  under¬ 
stood  the  effects  of  schedule  changes, 
either  slips  or  compressions.  Data  analy¬ 
sis  indicated  that  12  respondents  (48  per¬ 
cent)  thought  that  there  were  no  techniques 
or  processes  in  place,  or  that  the  assess¬ 
ment  of  schedule  change  effects  were 
based  only  on  subjective  inputs.  Several 
comments  highlight  the  lack  of  techniques 
or  processes  used  in  understanding  the 
effects  of  weapon  systems  schedule 
changes: 

Everyone  understands  the  effects 
of  slipping  schedule,  but  we  do  it 
anyway  and  don’t  quantify  the 
results. 

I  don’t  think  there  are  tools;  I  don’t 
even  think  it’s  a  process.  There  is 
no  process. 

No  real  tools  used.  Usually  based 
on  subjective  assessment  brought 
up  by  the  User  or  the  Program 
Element  Monitor. 


Additional  Concepts  of  Interest 

The  following  five  sections  address 
several  additional  concepts  of  interest  that 
were  identified  during  data  analysis  — 
failure  to  integrate,  lack  of  program  kill 
points,  cultural  aspects,  big  versus  small 
programs,  and  budgetary  focus.  These 
issues,  while  unsolicited,  are  important  in 
accurately  describing  the  current  AFCS 
decision-making  environment. 

Failure  to  Integrate.  The  integrated 
environment  described  as  a  unique  factor 
that  distinguished  weapon  systems  devel¬ 
opment  from  other  commercial  NPD 
activities  was  discussed  earlier.  However, 
12  respondents  (48  percent)  expressed 
concern  regarding  their  perception  that  the 
current  processes  do  not  take  a  holistic 
view  into  consideration  when  making 
weapon  systems  resource  allocation 
decisions.  A  respondent  stated: 

As  far  as  I  can  tell  there  are  no 
models  that  show  those  pieces  of 
the  Air  Force  system  and  what  the 
effect  of  what  one  part  of  the  sys¬ 
tem  has  on  the  entire  system. . . .  Too 
often  our  decisions  are  based  on 
how  does  this  impact  the  program, 
not  what  does  this  do  to  the  overall 
DoD  and  Air  Force  mission. 

A  decision  support  respondent  described 
decisions  being  made  in  a  vacuum: 

We  tend  to  measure  value  in  a 
vacuum.  In  other  words,  in  this 
one  mission  area  what’s  the  value 
of  it,  and  they  usually  compare  it 
to  something  that  has  existed  be¬ 
fore....  So,  when  you  measure 
what  the  value  of  a  weapon  sys¬ 
tem  is,  we  take  it  in  a  vacuum, 
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we  don’t  look  at  the  rest  of  the 
stuff  that  needs  to  be  brought 
along  with  it. 


Lack  of  Program  Kill  Points.  Nearly 
half  of  those  interviewed  described  a 
development  cycle  where  once  a  devel¬ 
opment  project  begins,  it  rarely  is  consid¬ 
ered  for  cancellation.  Twelve  of  those  in¬ 
terviewed  (48  percent)  address  several 
drivers  behind  keeping  programs  alive 
even  when  a  program  may  be  behind 
schedule,  overrunning  costs,  and  actual  de¬ 
velopment  progress  is  minimal.  These  driv¬ 
ers  included  political  influence  and  the  de¬ 
fined  requirement  for  the  weapon  system. 
In  addressing  the  requirement  issue,  a  re¬ 
spondent  commented: 


The  problem  is  that  we  just  can’t 
kill  a  program  if  the  need  or  re¬ 
quirement  still  exists.  We  have  to 
look  for  different  ways  to  solve 
this  need. ...  I  haven’t  seen  many 
things  killed. 


Another  of  those  interviewed  stated: 


We  rarely  kill  a  program;  we  just 
stretch  programs  out  because 
most  programs  are  justified  based 
on  a  mission  need  and  require¬ 
ment.  The  fact  that  we  can’t  af¬ 
ford  it  doesn’t  take  away  from  the 
valid  need  and  requirement  for  the 
system. 


Sure,  there  are  people  that  want  to 
kill  programs,  but  there  are  jobs  at 
stake,  Congressional  districts  at 
stake,  and  there  are  lots  of  avenues 
to  get  decisions  turned  around. 

Cultural  Aspects.  Program  advocacy, 
risk  averseness,  and  decision  maker  ex¬ 
perience  all  were  identified  to  some  ex¬ 
tent  by  those  in¬ 
terviewed  as 
components  to 
the  current 
AFCS  decision 
making  process. 

Discussion  on 
program  advo¬ 
cacy  centered 
on  the  perception  that  rather  than  playing 
the  role  of  “honest  brokers,”  many  in¬ 
volved  with  the  AFCS  process  were  more 
concerned  with  securing  their  program’s 
funding  stream.  Regarding  the  role  of  pro¬ 
gram  advocacy,  one  of  the  14  respondents 
(56  percent)  described  this  environment, 
stating: 


"The  problem  is 
that  we  just  can't 
kill  a  program  if 
the  need  or 
requirement 
still  exists." 


Every  program  has  a  constituency 
and  no  matter  how  bad  it’s  doing 
it’s  real  difficult  to  turn  it  off.  The 
whole  emphasis  is  upon  getting 
your  program  funded.  If  you  can 
get  your  program  funded  at  40  per¬ 
cent,  that’s  ok,  you  are  in  the 
game  and  you  can  fight  for  future 
funding  to  get  well. 


A  respondent  described  the  effects  poli¬ 
tical  influence  can  have  when  making 
decisions  regarding  whether  a  program 
gets  cut.  He  commented: 


Seven  of  those  interviewed  (28  percent) 
described  a  system  that  seems  to  be  risk 
averse  in  nature.  Concerns  raised  included 
the  perception  that  risk  and  uncertainty 
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within  development  programs  is  down¬ 
played  and  the  belief  that  a  clear  end 
product  must  be  defined  prior  to  allocation 
of  resources.  Further  insight  is  provided 
by  a  respondent  who  stated: 

From  an  overall  view,  the  AFCS 
process  is  risk  averse.  In  other 
words,  if  you  put  forward  a  pro¬ 
gram  or  an  initiative  that  says,  “If 
you  invest  this  much  money,  you 
have  this  chance  of  getting  this 
return.”  The  response  is  that  if 
you  are  unsure  about  your  suc¬ 
cess,  you  must  not  need  this  money 
very  much. 

Finally,  10  respondents  (40  percent) 
addressed  the  importance  that  the  ex¬ 
perience  level  of  the  decision  maker  plays 
in  the  AFCS  development  portfolio 
management  process. 

Big  Versus  Small  Programs.  Data 
analysis  indicated  that  many  respondents 
perceived  that  bigger  programs  have  a 
clearer  path  in  securing  development 
resources  and  may  in  fact  hamper  the 
development  of  smaller  programs.  Nine  of 
those  interviewed  (36  percent)  described 
the  difference  between  programs  like  the 
F-22,  Joint  Strike  Fighter,  and  Airborne 
Laser  and  small  programs,  and  the  effect 
these  differences  had  on  the  AFCS  deci¬ 
sion-making  process.  A  policy  maker 
within  the  process  provided  additional 
insight,  stating: 

Sometimes  when  it  involves  pro¬ 
grams  that  are  near  and  dear  to 
what  the  core  of  the  Air  Force  is 
all  about,  in  particular  I’m  think¬ 
ing  F-22s,  the  premier  weapon 
systems.  Those  (programs)  get  a 


lot  of  scrutiny,  everyone  tries  to 
understand  the  ramifications  of 
those  programs.  There  is  more 
information  available,  more  in¬ 
formation  flowing.  The  smaller 
programs,  I’m  not  convinced 
corporately  they  get  much  insight. 

Budgetary  Focus.  Data  analysis  re¬ 
vealed  that  64  percent  (16)  of  those  inter¬ 
viewed  perceived  the  AFCS  decision¬ 
making  process  to  be  driven  by  budgetary 
issues  rather  than  specific  program  issues. 
When  asked  to  what  extent  decision  sup¬ 
port  tools  were  used,  a  decision  maker 
commented: 

Decision  support  tools  that  are 
used  today  just  check  to  make 
sure  that  you  have  gotten  to  the 
bottom  funding  line  and  are  within 
the  budget  allowed. 

Comparison  to  an  Ideal  Model  of 
Development  Portfolio  Management 

In  order  to  effectively  complete  the 
assessment  process,  it  is  imperative  that 
the  discoveries  made  during  the  data  analy¬ 
sis  process  —  the  common  threads  —  be 
compared  to  an  “ideal”  model  of  how  de¬ 
velopment  portfolios  are  selected  and  man¬ 
aged.  As  previously  identified,  past  re¬ 
search  efforts  have  focused  on  defining 
best  practices  for  both  commercial  NPD 
and  R&D  portfolio  management.  This  ef¬ 
fort  has  been  driven  by  the  need  to  take 
the  first  step  in  understanding  current  Air 
Force  weapon  systems  development  port¬ 
folio  management  practices.  The  work  of 
Cooper,  Edgett,  and  Kleinschmidt  (1995, 
1998a,  1998b,  1999,  2000)  has  focused 
primarily  on  defining  best  practices  for 
R&D  portfolio  management.  As  such,  their 
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Table  2.  Comparison  of  Weapon  Systems 
and  Commercial  Best  Practices 


Success  Factors 

Commercial  Best 
Practices 

Air  Force  Portfolio 
Management  Practices 

Strategic  Focus 

(Cooper  et  al.,  1998b;  Danila, 

1989;  De  Maio  et  al.,  1994;  Griffin, 

1997;  Hall  &  Naudia,  1990;  Lester, 

1998;  Liberatore,  1987;  Souder  & 
Mandakovic,  1986;  Weber  et  al.,  1990; 
Wheelwright  &  Clark,  1992) 

Corporate  goals,  objectives,  and 
strategies  must  be  the  basis  for 
portfolio  selection. 

Perception  that  no  process  in 
place  to  measure  project’s 
alignment  with  strategic  goals. 

Senior  Management 
Support 

(Calantone  et  al.,  1995;  Cooper  et  al., 
1998b;  Danila,  1989;  Gupta  et  al., 

2000;  Hall  &  Naudia,  1990;  Lester, 

1998;  Liberatore,  1987;  Poolton  & 
Barclay,  1998;  Scott,  2000) 

Senior  management  must  be 
closely  involved  in  R&D  project 
selection  decisions. 

Senior  leadership  support 
perceived  as  a  must  for 
development  project  survival. 

Communication 

(Cooper  et  al.,  1998b;  Danila,  1989; 
Gupta  et  al.,  2000;  Poolton  &  Barclay, 
1998;  Souder  &  Mandakovic,  1986) 

Must  establish  good  communi¬ 
cation  links,  both  internal  and 
external. 

Multiple  established  paths  for 
flow  of  strategic  policy 
information. 

Portfolio  Methods 
and  Use 

(Cooper  et  al.,  1998b;  De  Maio  et  al., 
1994;  Hall  &  Naudia,  1990;  Weber  et 
al.,  1990) 

Multiple  methods  incorporated 
—  used  to  gain  insight  into  each 
project. 

Little  to  no  decision  support 
processes  implemented. 

Primarily  budgetary  focus. 

Flexibility 

(Calantone  et  al.,  1995;  Cooper  et  al., 
1998b;  De  Maio  et  al.,  1994;  Scott, 
2000;  Weber  et  al.,  (1990); 

Wheelwright  &  Clark,  1992) 

Able  to  address  resource,  benefit, 
and  outcome  interactions.  Adapt 
to  changes  in  goals,  requirements, 
and  project  characteristics. 

Perceived  lack  of  assessment 
as  to  the  impacts  that  portfolio 
management  decisions  have  on 
other  weapon  systems. 

Decision  Making  at 
Different  Levels 

(Cooper  et  al.,  1998b;  Liberatore, 

1 987;  Wheelwright  &  Clark,  1992) 

Must  be  able  to  address  the 
organizational  structure  and  the 
decision-making  process  involved 
at  each  level. 

Not  directly  assessed.  Limited 
decision-making  information 
available  and  compressed 
decision-making  time  frame. 

Risk 

(Cooper  et  al.,  1 998b;  Wind  & 

Mahajan,  1988) 

Risk  and  uncertainty  must  be 
addressed  within  the  selection 
process. 

Perception  that  risk  and 
uncertainty  not  adequately 
addressed  during  portfolio 
management  process. 

Multi-functional  Approach 

(Calantone  et  al.,  1995;  Griffin,  1997; 
Gupta  et  al.,  2000;  Lester,  1998; 

Scott,  2000) 

Effective  implementation  of 
multi-functional  teams  in  the 
portfolio  management  process. 

Multi-function  evaluation  groups 
at  each  level  of  the  AFCS. 
Perception  of  stovepiping  at 
lower  levels  (Panels/IPTs). 

Customer  Focus 

(Calantone  et  al.,  1995;  Gupta  et  al., 
2000;  Lester,  1998;  Poolton  & 

Barclay,  1998;  Scott,  2000) 

Needs  and  requirements  of 
the  User  assessed  during  the 
project  selection  process. 

User’s  validated  need  or 
requirement  key  driver  in 
portfolio  management 
process. 
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findings  form  the  foundation  of  our  best 
practices  model.  Additional  study  of  past 
research  efforts  was  conducted,  and  con¬ 
cepts  and  success  factors  that  are  com¬ 
mon  among  the  literature  were  added  to 
our  best  practices  criteria  list.  The  con¬ 
cepts  of  process  multi-functional  approach 
and  customer  focus  were  added  to  this 
foundation  to  develop  an  “ideal”  model. 
Table  2  frames  the  comparison  between 
commercial  best  practices  and  the  results 
of  this  research. 

Several  similarities  were  discovered 
during  this  comparison  process.  First,  both 
commercial  and  weapon  systems  portfolio 
management  practices  place  great  value 
in  the  needs  and  requirements  of  the  User. 
Second  the  Air  Force  development  port¬ 
folio  management  process  is  structured  so 
that  multi-functional  representation  is  the 

common  envi¬ 
ronment  found 
at  each  level  of 
the  AFCS.  Fi¬ 
nally,  clear 
paths  for  the 
communication 
of  strategic 
policy  and  the 
importance  of 
gaining  senior 
leadership  support  for  a  weapon  systems 
development  project  align  closely  with 
those  identified  as  commercial  best  prac¬ 
tices. 

Perhaps  more  useful  in  beginning  the 
process  of  closing  the  gap  between  cur¬ 
rent  practices  and  commercial  best  prac¬ 
tices  is  to  identify  areas  where  there  are 
differences  between  the  two  development 
portfolio  management  models.  Several 
such  areas  were  exposed  during  this 
research.  Perceived  practices  including 


limited  use  of  decision  support  tools;  lack 
of  a  robust  process  to  assess  a  project’s 
alignment  with  strategic  goals  and  objec¬ 
tives;  an  environment  where  decision 
making  information  is  limited  and  the 
decision  making  time  frame  is  compressed; 
a  decision-making  process  that  is  perceived 
to  inadequately  assess  and  incorporate 
risk;  and  a  decision-making  process  that 
is  perceived  to  make  decisions  without  as¬ 
sessing  and  incoiporating  the  impacts  to 
other  weapon  systems  development 
projects  were  all  identified  as  areas  where 
the  practices  of  the  AFCS  do  not  align  with 
those  found  among  researched  commer¬ 
cial  industry  leaders. 

Discussion  and  Conclusion 


Acquisition  reform  efforts  have  been 
key  drivers  in  improving  Air  Force,  and 
DoD  in  general,  weapon  systems  devel¬ 
opment  and  procurement  processes  and 
practices.  Such  reform  efforts  have  called 
for  the  emulation  and  migration  toward  a 
more  commercial-like  approach  to  weapon 
systems  development  and  procurement 
activities.  The  findings  of  this  research 
effort  indicate  that  current  Air  Force 
development  portfolio  management 
practices  are  not  aligned  with  those  com¬ 
mercial  best  practices  described  in  the 
literature. 

While  this  research  focused  on  assess¬ 
ing  and  describing  the  “as-is”  environment 
of  Air  Force  weapon  systems  develop¬ 
ment  portfolio  management  practices,  re¬ 
search  results  should  provide  those  in  lead¬ 
ership  positions  with  a  foundation  from 
which  the  first  steps  toward  process  im¬ 
provement  can  be  identified  and  devel¬ 
oped.  Although  not  all  inclusive,  potential 


"...both 

commercial  and 
weapon  systems 
portfolio  manage¬ 
ment  practices  place 
great  value  in  the 
needs  and  require¬ 
ments  of  the  User." 
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process  improvement  paths  include  sepa¬ 
rating  the  review  and  evaluation  of  new 
initiatives  from  current  ongoing  programs 
and  other  Air  Force  resource  allocation 
activities,  refining  the  current  decision¬ 
making  support  process,  defining  methods 
for  increasing  personnel  stability  within  the 
decision-making  process,  and  researching 
alternative  methods  for  defining  and 
measuring  the  value  of  a  weapon  system. 


This  exploratory  research  has  provided 
insights  into  the  AFCS  development  project 
portfolio  management  decision-making 
process.  Developing  abetter  understanding 
of  current  Air  Force  development  portfo¬ 
lio  management  practices  and 
benchmarking  them  against  pertinent  com¬ 
mercial  best  practices  is  a  necessary  first 
step  to  begin  bridging  the  gap  between 
these  two  environments. 
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Appendix 

Interview  Questions 


Background  Information 

1 .  What  is  your  current  duty  title? 

2.  How  long  have  you  been  in  your  current  position? 

3.  What  is  your  role  in  the  weapon  systems  development  project  selection  process? 

Strategic  Alignment 

4.  What  processes  are  in  place  to  determine  a  potential  development  project’s  align¬ 
ment  with  these  strategic  goals  and  objectives  at  the  DoD  level,  the  Air  Force  level, 
the  MAJCOM/CINC  level,  and  the  Weapon  Systems  level? 

5.  Please  assess  the  following  statement:  the  AFCS  decision-making  process  ensures 
that  a  weapon  systems  development  project’s  capabilities  align  with  the  strategic 
goals  and  objectives  at  the  DoD  level,  the  Air  Force  level,  the  MAJCOM/CINC 
level,  and  the  Weapon  Systems  level.  * 

Communication 

6.  By  what  means  are  the  different  levels  of  strategic  goals  and  objectives  communi¬ 
cated  to  decision  makers  in  the  AFCS  decision-making  process? 

7.  Describe  the  types  of  information  made  available  to  decision  makers  in  the  AFCS 
regarding  the  specific  development  projects  under  consideration. 

8.  Based  on  your  assessment,  is  the  information  provided  to  decision  makers  based 
more  on  objective  analysis,  or  is  it  based  more  on  subjective  assessments ? 

9.  Based  on  your  assessment,  what  percentage  of  new  development  projects  enters 
the  AFCS  decision-making  process  via  the  bottom-up  approach  versus  those  that 
enter  the  process  as  an  initiative  directed  by  senior  leadership? 

10.  Please  assess  the  following  statement:  from  an  overall  perspective,  the  AFCS  deci¬ 
sion-making  process  provides  effective  and  efficient  opportunities  for  the  flow  of 
development  project  information  within  the  same  evaluation  group  and  between 
different  evaluation  groups.  * 
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Decision  Support  Tools 

1 1 .  In  the  AFCS  decision-making  process,  what  types  of  decision  support  tools  are  used 
during  the  evaluation  and  selection  of  weapon  systems  development  projects? 

12.  Based  on  your  assessment,  are  the  decision  support  tools  that  are  implemented  used 
more  for  generating  optimized  project  ranking  lists,  or  are  they  used  more  for 
providing  insight  into  each  development  project  alternative ? 

13.  Please  assess  the  following  statement:  decision  support  tools  are  used  what  percent¬ 
age  of  the  time  in  the  AFCS  decision-making  process  when  evaluating  and  selecting 
weapon  systems  development  projects.  * 

Feedback 

14.  Once  a  weapon  systems  project  has  initially  entered  the  development  process,  and 
returns  to  the  AFCS  for  recurring  review,  what  types  of  feedback  are  provided  to 
the  decision  maker  regarding  the  current  status  of  the  development  effort? 

15.  Please  assess  the  following  statement:  the  AFCS  decision-making  process  provides 
effective  and  efficient  opportunities  for  feedback  to  occur  regarding  the  current 
status  of  a  weapon  systems  development  project  under  review.  * 

Risk  and  Uncertainty 

16.  Identify  sources  of  risk  and  uncertainty  within  the  AFCS  and  discuss  the  effects  that 
risk  and  uncertainty  have  on  decision  makers  and  the  decision-making  process. 

17.  Identify  the  tools  and  techniques  currently  implemented  within  the  AFCS  that  pro¬ 
vide  initial  risk  and  uncertainty  assessments  inherent  in  a  weapon  systems  develop¬ 
ment  project. 

18.  Please  assess  the  following  statement:  the  AFCS  decision-making  process  ensures 
that  risk  and  uncertainty  are  identified  and  assessed  during  the  weapon  systems 
development  project  selection  process.  * 


Value  Determination 

19.  Identify  the  tools  and  techniques  currently  implemented  within  the  AFCS  that  are 
used  to  determine  a  weapon  systems  development  project’s  value. 

20.  Please  assess  the  following  statement:  the  AFCS  decision-making  process  ensures 
that  a  weapon  systems  development  project’s  value  is  assessed  and  considered  in 
the  decision-making  process.  * 
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Schedule  Effects 

21.  Identify  the  tools  and  techniques  currently  implemented  within  the  AFCS  that  are 
used  to  determine  the  effects. 

22.  Please  assess  the  following  statement:  the  AFCS  decision-making  process  ensures 
that  the  effects  of  schedule,  either  compression  or  slips,  is  assessed  and  considered 
in  the  weapon  systems  development  project  decision-making  process.  * 

Strengths  and  Weaknesses 

23.  What  are  some  additional  strengths  of  the  Air  Force’s  current  process  for  evaluating 
and  selecting  weapon  systems  development  projects? 

24.  What  are  some  additional  weaknesses  of  the  Air  Force’s  current  process  for  evalu¬ 
ating  and  selecting  weapon  systems  development  projects? 


*  FIVE-POINT  LIKERT  SCALE  RESPONSE 
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